Skip to content

Multiple file input #29

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 43 commits into
base: master
Choose a base branch
from
Open

Multiple file input #29

wants to merge 43 commits into from

Conversation

jay-tau
Copy link
Collaborator

@jay-tau jay-tau commented May 1, 2025

No description provided.

jay-tau added 30 commits March 17, 2025 22:59
…and stragglers count in Darshan data handling
jay-tau added 13 commits April 28, 2025 14:22
Avoids using `iterrows()` which implicitly casts Darshan record IDs to float64,
potentially breaking file_map lookups for large IDs. Replaced with `itertuples()`
to maintain original dtype and re-enabled `break` for early exit.
…e logs

Adds Sample_1 and Sample_2 run configs to easily test `reporter.py` on
benchmark Darshan logs using the drishti-io module environment.
@jay-tau jay-tau marked this pull request as ready for review June 18, 2025 23:29
@jay-tau jay-tau requested review from Copilot and jeanbez and removed request for Copilot June 18, 2025 23:30
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR updates the project to support multiple file inputs by changing the parser argument from a single log path to a list of paths. It also refactors several modules—reporter, parser, and the Darshan handler—to utilize the new argument, adds development tool dependencies, and introduces various assertions for metric validation.

  • Updated parser: renamed “log_path” to “log_paths” with nargs="+"
  • Refactored reporter and handle_darshan to work with multiple log inputs and utilize a DarshanFile object
  • Added dev tools extras_require in setup.py and updated several formatting and assertion checks in the handler

Reviewed Changes

Copilot reviewed 5 out of 17 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
setup.py Added dev dependencies under extras_require
drishti/reporter.py Updated log type detection to work with lists and adjusted import patterns
drishti/includes/parser.py Changed positional argument from "log_path" to "log_paths" with improved flag formatting
drishti/handlers/handle_darshan.py Refactored handler to use DarshanFile, added assertions, and updated module check function calls
.python-version Set Python version to 3.8
Files not reviewed (9)
  • .idea/.gitignore: Language not supported
  • .idea/MypyPlugin.xml: Language not supported
  • .idea/drishti-io.iml: Language not supported
  • .idea/inspectionProfiles/profiles_settings.xml: Language not supported
  • .idea/misc.xml: Language not supported
  • .idea/modules.xml: Language not supported
  • .idea/runConfigurations/Sample_1.xml: Language not supported
  • .idea/runConfigurations/Sample_2.xml: Language not supported
  • .idea/vcs.xml: Language not supported
Comments suppressed due to low confidence (1)

drishti/handlers/handle_darshan.py:572

  • The call to sys.exit(2) before invoking the shared small operation check may lead to premature termination of the handler. Verify if this exit is intentional; if not, consider removing it or using proper error handling.
            module.check_shared_small_operation(total_shared_reads, total_shared_reads_small, total_shared_writes, total_shared_writes_small, shared_files, file_map)

Comment on lines +44 to +52
is_darshan = True and is_darshan
is_recorder = False and is_recorder
else: # check whether is a valid recorder log
if not os.path.isdir(path):
print("Unable to open recorder folder.")
sys.exit(os.EX_NOINPUT)
else:
is_recorder = True and is_recorder
is_darshan = False and is_darshan
Copy link
Preview

Copilot AI Jun 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The use of expressions like 'True and is_darshan' and 'False and is_recorder' is redundant. Consider assigning the intended boolean value directly (e.g. 'is_darshan = True', 'is_recorder = False') to improve code clarity.

Copilot uses AI. Check for mistakes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant